Privacy by Design
in the machine learning pipeline


Workshop, University of Venice

Zak Varty

2024-11-01

Hello! Who am I?

  • Statistician by training, got to where I am through medical, environmental and industrial applications.

  • Teaching Fellow at Imperial College London

    • Data Science, Data Ethics

  • Privacy, fairness and explainability in ML.

    • Capable enthusiast, realist / pessimist

Really it all comes down to doing good statistics well.


Why do we care about privacy?

  1. Protecting Sensitive Data: ML models often train on personal or confidential data (health records, financial info), and safeguarding this data is essential to prevent misuse or unauthorized access.

  2. Compliance with Regulations: Laws like GDPR require organisations to protect user privacy, impacting how data is collected, stored, and used in ML.

  3. Preventing Data Leakage Models can unintentionally expose sensitive information from their training data, risking user privacy if someone exploits the model’s outputs.

  4. Building Trust: Privacy-conscious ML practices foster trust among users, making them more willing to share data and participate in systems that use ML.

  5. Avoiding Discrimination: Privacy techniques can reduce bias and discrimination risks, ensuring the ML model treats users fairly without targeting sensitive attributes.

“If you have nothing to hide, you have nothing to fear”

“The benefits outweight the costs”

Machine Learning Pipeline

Machine Learning Pipeline Life Cycle

Reality is much messier. See Gelman and Loken (2013) for a discussion of the implications.

This workshop

  • Work through the ML life cycle
  • How could and have things gone wrong


  • What tools do we have?
  • What are their limitations?


  • Interactive bits, every now and then

1. Data Collection

Collecting Appropriate Data - GDPR

  • Collect only necessary data, purpose clear at time of collection
  • Data subject has right to
    • access
    • rectify
    • erase
    • object
  • Consequences both financial and reputational

Collecting Data - Hard-to-Reach Groups

Standard ML assumes that data are cheap and easy to collect.

Assumption that data are cheap and easy to collect. Out of the box model fitting assumes we are working with big, representative and independent samples.

  • Applications of ML to social science to study hard-to-reach populations: persecuted groups, stigmatised behaviours.

  • Standard study designs and analysis techniques will fail.

Snowball Sampling - Hidden Network

Snowball Sampling - Initial Recruitment

Snowball Sampling - Referral Round 1

Snowball Sampling - Referral Round 2

Snowball Sampling - Referral Round 3

Snowball Sampling

By using subject-driven sampling design, we can better explore the hard to reach target population while preserving the privacy of data subjects who do not want to be included in the study.

  • Bonus: Also allows us to study community (network) structure if we are interested in that.
  • Drawbacks:
    • “isolated” nodes,
    • partitioned graphs,
    • adapting model fitting to non-uniform sampling.

Collecting Data - Asking Difficult Questions

Even if data subjects are easy to access and sample from, they may not wish to answer honestly.

Can you give me some examples?

  • cheated on an exam?
  • cheated on a romantic / sexual partner?
  • experienced suicidal ideation?
  • killed another person?

Dishonest answers will bias any subsequent analysis, leading us to underestimate the prevalence of an “undesirable outcome”.

(Interesting intersection with survey design and psychology. The order and way that you ask questions can influence responses but we will focus on a single question here.)

Direct Response Survey

\[\Pr(Y_i = 1) = \theta \quad \text{and} \quad \Pr(Y_i = 0) = 1 - \theta.\] Method of Moments Estimator: (General dataset)

\[ \hat \Theta = \hat\Pr(Y_i = 1)\]

\[ \hat \Theta = \frac{1}{n}\sum_{i=1}^n \mathbb{I}(Y_i = 1) = \bar Y = \frac{\#\{yes\}}{\#\{subjects\}}.\]

Method of Moments Estimate: (Specific dataset) \[ \hat \theta = \frac{1}{n}\sum_{i=1}^n \mathbb{I}(y_i = 1) = \bar y.\]

MoM Example

Suppose I ask 100 people whether they have ever been unfaithful in a romantic relationship and 24 people respond “Yes”.


What is your best guess of the proportion of all people who have been unfaithful?

\(\hat\theta = \bar y = \frac{24}{100}\)


How confident are you about that guess?

Would that change if I had 1 person responding “Yes”?

Would that change if I had 99 people responding “Yes”?

MoM - Nice Properties

Over lots of samples we get it right on average:

\(\mathbb{E}_Y[\hat\Theta] = \mathbb{E}_Y\left[\frac{1}{n}\sum_{i=1}^n \mathbb{I}(Y_i = 1)\right] = \frac{1}{n}\sum_{i=1}^n \mathbb{E}_Y\left[ \mathbb{I}(Y_i = 1)\right] = \frac{n \theta}{n} = \theta\)

As the number of samples gets large, we get more confident and therefore recover the truth

\[\begin{align*} \text{Var}_Y[\hat\Theta] &= \text{Var}_Y\left[\frac{1}{n}\sum_{i=1}^n \mathbb{I}(Y_i = 1)\right] \\ &= \frac{1}{n^2}\sum_{i=1}^n\text{Var}_Y\left[\mathbb{I}(Y_i = 1)\right] \\ &= \frac{1}{n^2}\sum_{i=1}^n p(1-p) \\ &= \frac{n p (1-p)}{n^2} \\ &= \frac{p (1-p)}{n} \rightarrow 0 \end{align*}\]

Adding Privacy - Randomised Response

  • Mathematically nice but in reality people lie.

  • Our estimator worked “best” for central values of \(\theta\), unlikely for stigmatised events.

  • Add random element to survey to provide plausible deniability.

    • Flip a fair coin. If heads, switch your answer.

  • MoM estimation: Equate probabilities and proportions.

Estimation from Randomised Response Data

Consider using a weighted coin, probability \(p\) of telling truth. Derive an expression for the probability of answering “Yes”.

\[\begin{align*} \Pr(\text{Yes}) &= \theta p + (1 - \theta)(1 - p) \\ & \approx \frac{\#\{yes\}}{\#\{subjects\}} \\ &= \bar y \end{align*}\]

Rearrange this expression to get a formula for \(\hat \theta\).

\[\hat \theta = \frac{\bar y - 1 + p}{2p -1}.\]

Randomised Response Activities

\[ \hat \theta = \frac{\bar y - 1 + p}{2p -1}.\]

  • Direct response is a special case of randomised response. How can we use that to check our working?
  • If our previous survey results came from this randomised response survey design with \(p = 0.25\), what is your best guess of the proportion of people who have been unfaithful?
  • Are you more or less confident in this estimate than previously, when we had the same data but from a direct response design?
  • What does your confidence depend on? And which of those factors do you have knowledge of / control over?
  • When would this estimation procedure break and why? How could we fix that?

Randomised Response - Privacy Schematic

Randomised Response

  • Approach to privacy for single, binary response.

  • Issues with applying to multiple questions, e.g. surveys with follow on questions.

  • Extensions to categorical and continuous responses and predictors

  • General principle of adding noise

    • Need to make observations indistinguishable
    • Need to preserve important aspects of “signal”

Data Collection Summary

  • Collect what you need and use that information only for its intended purpose.

  • Targeting hard-to-reach populations can be challenging but possible by combining survey design and specific learning approaches. Keeps statisticians in a job!

  • Asking difficult questions can lead to biased responses. Plausible deniability through randomised response designs can help.

2. Data Storage

Encryption

Once we have gone to the effort of collecting data we don’t want to just leave it lying around for anyone to access.


\[ \text{Plain text} \overset{f}{\rightarrow} \text{Cipher Text}\]
\[ \text{Cipher text} \overset{f^{-1}}{\rightarrow} \text{Plain Text}\]
\[ f(\text{data}, \text{key})\] Many encryption schemes depending on the data to be encrypted and how the key is to be distributed.

Caesar Cipher

Write your own short message and pass it to a friend to decode.

What are some benefits and drawbacks of this encryption scheme?

K-anonymity

What happens if someone gets access to the data?


  • \(k\)-anonymity is a measure of privacy within a dataset.

  • Given a set of predictor-outcome responses, each unique combination forms an equivalence class.

  • The smallest equivalence class of a \(k\)-anonymous dataset is of size \(k\).

  • Equivalently, each individual is indistinguishable from at least \(k-1\) others.

K-anonymity Example

K-anonymity Worked Example

K-anonymity Your Turn

I asked ChatGPT to generate 4-anonymous datasets but it hasn’t done a good job.

  • Establish the true value of k for your dataset.

  • Use pseudonymisation, aggregation, redaction and partial-redaction to make your dataset 4-anonymous.

K-anonymity Feedback

  • How did ChatGPT do?
  • How did you alter the anonymity level of your dataset?
  • What did you have to consider as you were doing this?

K-anonymity Drawbacks

What do you think some of the limitations of \(k\)-anonymity might be?

  • Knowing what is important before analysis.
  • Publishing multiple versions of the dataset.
  • Can be checked easily but not implemented algorithmically.
  • External data attacks; Jane Doe, Latanya Sweeney

Data Storage - Summary

  • Don’t leave important data lying around unprotected.

  • Choose a level of security appropriate to the sensitivity of the data.

  • Consider the consequences of someone gaining access to the data.

  • Remember that your data does not live in isolation.

  • K-anonymity not a good measure of privacy but an accessible starting point.

End of Part 1
Tomorrow: analyitcs, modelling and production.

3. Data Analytics

4. Modelling

5. Going into Production

Build Information

R version 4.3.3 (2024-02-29)

Platform: x86_64-apple-darwin20 (64-bit)

locale: en_US.UTF-8||en_US.UTF-8||en_US.UTF-8||C||en_US.UTF-8||en_US.UTF-8

attached base packages: stats, graphics, grDevices, utils, datasets, methods and base

loaded via a namespace (and not attached): compiler(v.4.3.3), fastmap(v.1.1.1), cli(v.3.6.3), tools(v.4.3.3), htmltools(v.0.5.8.1), rstudioapi(v.0.16.0), yaml(v.2.3.8), Rcpp(v.1.0.12), pander(v.0.6.5), rmarkdown(v.2.26), knitr(v.1.45), jsonlite(v.1.8.8), xfun(v.0.43), digest(v.0.6.35), rlang(v.1.1.4), png(v.0.1-8) and evaluate(v.0.23)

References

Privacy by Design - November 2024 - Zak Varty

Gelman, Andrew, and Eric Loken. 2013. “The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No ‘Fishing Expedition’ or ‘p-Hacking’ and the Research Hypothesis Was Posited Ahead of Time.” Department of Statistics, Columbia University 348 (1-17): 3.